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Experimental program review in our field may benefit from a more quantitative framework witliin 
wliicli to quantitatively discuss tfie scientific merit of a proposed program of researdi, and to as- 
sess the scientific merit of a particular experimental result. This article proposes explicitly such 
a quantitative framework. Examples of the use of this framework in assessing the scientific merit 
of particular avenues of research at the energy frontier in many cases provide results in stark con- 
tradiction to accepted wisdom. The experimental scientific figure of merit proposed here has the 
potential for informing future choices of research direction in our field, and in other subfields of the 
physical sciences. 



In such a quantitative field as experimental high energy 
physics, the discussions leading to these decisions and 
evaluations are notably non-quantitative. 

The intent of Sec.|TT]is to suggest a quantitative frame- 
work for such discussions, including a specific figure of 
merit to quantify the scientific merit of a proposed exper- 
imental research program, and a specific figure of merit to 
quantify the scientific merit of a completed experimental 
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result. Sec. Ill provides explicit worked examples. 



I. MOTIVATION 

In the context of determining which research programs 
to pursue, review committees often must decide the rel- 
ative scientific merits of proposed experiments. Within 
large experiments, deciding which analyses to emphasize 
requires similar decisions. These issues arise in the dis- 
cussion of experiments or analyses in which the result is 
not yet known. 

A related issue is faced by those assessing the scientific 
merit of an experiment or analysis whose result is known. 
This topic is the subject of much innocuous lunchroom 
conversation, and more seriously in the evaluation of the 
organizations and individuals responsible for producing 
the result. 



II. INFORMATION CONTENT 

Given the academic and esoteric nature of our field, the 
appropriate figure of merit is not expected economic pay- 
off, but rather should be formulated in terms of an exper- 
iment's information content. Given the rather grandiose 
nature of our field, we will not concern ourselves with 
distinguishing qualitatively equivalent but quantitatively 
different states of understanding. Whether a Standard 
Model Higgs boson exists or not are considered two qual- 
itatively different states of understanding; if a Standard 
Model Higgs boson exists, whether it has mass 115 GeV 
or 160 GeV are considered two qualitatively equivalent 
states of understanding. 

The appropriate figure of merit to measure an experi- 
ment's information content is well known in the context 
of information theory. The original reference is Shan- 
non's classic 1948 paper, "A Mathematical Theory of 
Communication" [IJ. The essential idea behind this fig- 
ure of merit is developed in Sec.|II A" 



ticated version developed in Sec. |IIB 



with a more sophis- 
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A. Basic formulation 

The value of a particular experimental result in an aca- 
demic field clearly should be measured by how much is 
learned from the result. Equivalently, the value of a re- 
sult is how surprised you are that the particular result 
has been obtained. 

Suppose an experiment can have one of two out- 
comes. Denote the two possibilities by A = {xi,X2}, 
and let p{xi) and p{x2) represent the expectation of the 
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two possible outcomes xi and X2 being realized, with 
p{xi) + p{x2) = 1. For example, assigning p{xi) — 0.8 
and p{x2) = 0.2 represents betting odds of 4:1 in favor 
of outcome xi being obtained. In general, an experiment 
will have n possible outcomes X — xi, . . . , Xi, . . . , Xn, 
with each outcome Xi having expectation p{xi). 

The surprise of obtaining the particular result Xi is 
clearly large if p{xi) is small. The scientific value of the 
result Xi is therefore large \i p{xi) is small. The scientific 
value of the result Xi is small if p{xi) is near unity, since 
in this case little has been learned. The scientific value 
of obtaining the result Xi must thus be a monotonically 
decreasing function oip{xi). 

A good measure of scientific merit should have the de- 
sired property of addition: the scientific merit of a paper 
announcing the discovery of a Standard Model Higgs bo- 
son plus the scientific merit of a paper announcing the 
discovery of a new Z' boson should equal the scientific 
merit of a paper containing both discoveries. Equiva- 
lently, the act of stapling two papers together should not 
change their total scientific merit. 

The scientific merit of a particular result Xi should 
thus (i) be a monotonically decreasing function of the 
expectation p{xi) that the result would be obtained, and 
(ii) be appropriately additive. This leads uniquely to the 
choice [T] 



surprisal(xi) = ~\og^Qp{xi) 



(1) 



for the scientific merit of obtaining the result Xi. The 
negative sign in Eq. [l]is required for the scientific merit to 
be a monotonically decreasing function oip{xi), and the 
logarithm is required for the scientific merit to have the 
desired behavior under addition. The use of logarithm 
base ten is arbitrary but convenient, and corresponds to 
a choice of units [3T] [35] . 

This article suggests that the surprisal of a particular 
experimental result is the appropriate measure of scien- 
tific merit for an experiment or analysis whose result is 
known. The larger the surprisal of a particular outcome, 
the greater its (a posteriori) scientific merit. 

A measure of the scientific merit of an experimental 
result has thus been obtained in Eq. [T] The definition 
of rational behavior is decision making that maximizes 
expected payoff. The payoff of obtaining any particular 
result Xi is given by Eq.[l] The expected payoff is thus the 
payoff of each possible result Xi, weighted according to 
the expectation that the result Xi will be obtained. The 
expected payoff of an experiment with possible outcomes 



X = xi,.. 



, a;„ is thus 



H{X) = y^p(a;^)surprisal(a;^) = - '^p{xi)\og-^Qp{xi). 

i i 

(2) 

The expected payoff H{X) is thus the appropriate scien- 
tific merit of a proposed experiment, before the result is 
known. 

Nearly all of the discussion so far can be found in the 
first few pages of any introductory information theory 



text. The only new idea in this article is that the sur- 
prisal and expected surprisal of Eqs. [T| and [2] can actu- 
ally be applied in practice to quantify the scientific merit 
of completed and proposed high energy physics experi- 
ments. 



B. Advanced formulation 

The previous section introduced the essential idea of 
this article. This section completes the development of 
necessary concepts. 

In the previous section it was implicitly assumed that 
each possible experimental outcome corresponds one-to- 
one with a qualitatively new state of knowledge. This is 
clearly not always the case: flipping a coin can result in 
heads or tails, and thus obtaining heads has a nonzero 
surprisal according to Eq. [T] but obtaining heads does 
not reveal much about how Nature works. 

It is thus useful to consider a set of qualitatively 
distinct and mutually incompatible states of knowledge 
y = {yi y™}, the j'^ of which is generally ac- 
cepted to be correct with probability p{yj), in the usual 
Bayesian interpretation. Since the process of normal sci- 
ence relies to a significant degree on a scientific commu- 
nity's shared view of important problems and possible so- 
lutions, in practice there is little difficulty defining Y 
The possible states of knowledge are complete and mutu- 
ally exclusive, so that Yl^^iPiVj) — 1 An example 
might be y = {2/1,2/2}, where yi corresponds to the state 
of knowledge in which a Higgs boson is known to exist, 
and j/2 corresponds to a state of knowledge in which a 
Higgs boson is known not to exist. An experiment may 
have possible outcomes X — {xi,X2}, where xi corre- 
sponds to the result that a Higgs boson is found, and X2 
corresponds to the result that no Higgs boson is found. 
Depending on the sensitivity of the experiment, xi and 
X2 may not be in one-to-one correspondence with yi and 
?/2- A null result from a Higgs boson search at the Teva- 
tron, which is limited by the amount of data collected 
and the collider energy, will not prove the Higgs boson 
does not exist. 

A slightly more sophisticated understanding of 
Sec. II A is required. Noting that what counts is the 



state of knowledge rather than the particular experimen- 
tal outcome, Eq.|2]should be written in terms oiY rather 
than X: 



H{Y) = -Y,p{y,)\og^^p{y,). 



(3) 



The quantity H(Y) can be considered to be the entropy 
associated with the state of knowledge Y. The state of 
knowledge F is a superposition of orthogonal microstates 
yj, each with expectation p{yj). It was implicitly as- 
sumed in Sec. II A that the experiment under consid- 
eration results in a definitive state of knowledge, where 
one microstate yj is selected, corresponding to zero infor- 
mation entropy after the result is obtained. In general. 
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an experiment may result in a state of knowledge with 
nonzero information entropy: if the Higgs boson is not 
found at the Tevatron, the resulting state of knowledge 
will still consist of a superposition of the two possibilities 
that the Higgs boson does or does not exist, with slightly 
adjusted relative expectations. 

Assume the experimental result Xi is obtained. The 
expectation that j/j is correct was p{yj) before the ex- 
periment, and is p{yj\xi) after the experiment [51] . The 
information entropy after the experimental result Xi is 
obtained will be 

H{Y\x^) = -^p(yj|xi)logioP(2/j|a;*), (4) 

3 

in analogy with Eq. |3] 

Before the experiment is performed, it is not known 
what result will be obtained. Before the experiment is 
performed, the expected information entropy that will 
remain after the experiment is performed is 

H{Y\X) = J2pi^dH{Y\x,) 

i 

= - XI P^^'^ |a;») logio ki) 

i 3 

= -Yp{x^,y3)^O&10P{yj\x^)■ (5) 

Performing an experiment X is thus expected to reduce 
the information entropy of our state of knowledge Y from 
H{Y) to the conditional entropy H{Y\X). 

Before the experiment is performed, the information 
entropy H{Y) quantifies our degree of uncertainty. Per- 
forming the experiment X is expected to reduce this un- 
certainty to a smaller information entropy H(Y\X). The 
information expected to be gained by performing the ex- 
periment X is thus 

AH ^ H{Y) - H{Y\X). (6) 

This article suggests that the expected information 
content in a particular experiment or analysis, measured 
in terms of the expected reduction in information entropy 
AH, is the appropriate measure of scientific merit for an 
experiment or analysis whose result is not yet known. 
The larger the expected information content of an exper- 
iment, the greater its (a priori) scientific merit. 

Experiments that are guaranteed to result in a state of 
knowledge of zero entropy have H{Y\X) = 0. For such 
experiments, the information content AH is equal to the 
entropy H{Y) of the initial state of knowledge. 

Assume the state of knowledge yj is actually correct. 
The scientific merit of obtaining a particular result Xi 
should then be the additional evidence 

evidence(?/j \x^) = log^o ^^^^^^'^ (7) 

PiVj) 

that the experiment can provide in favor of the (correct) 
state of knowledge yj . The logarithm again ensures the 



desired behavior under addition. As an example, p{yj) = 
0.09 and p{yj\xi) = 0.90 results in evidence = 1, 
and the experimental result Xi corresponds to one unit 
of evidence in favor of yj . 

Since in practice the state of knowledge yj is not known 
to be correct, the scientific merit of obtaining the exper- 
imental result Xi is the evidence the result Xi provides in 
favor of yj , averaged over possibly correct states of knowl- 
edge yj, weighted according to the new expectation that 
each yj is correct: ^ [55] 

information gain (xi) = p{yj \xi) evidence(yj [x^) 

3 

The scientific merit of the experimental result Xi, quan- 
tified by Eq. [8] and motivated in terms of a weighted 
average of expected evidence, is the appropriate recast- 
ing of Eq. [T| in terms of the information gained on the 
state of knowledge Y from the experimental result Xi. 
The information gain defined in Eq. |8] is also sometimes 
referred to as information divergence, relative entropy, 
or the KuUback-Leibler divergence [5] in the fields of 
probability theory and information theory, and the max- 
imization of this utility function is referred to as Bayes 
d-optimal experimental design [3]. Intuitively, the in- 
formation gain is a measure of how much our state of 
knowledge has changed as a result of obtaining the out- 
come Xi- Eq. [8] reduces to the previous Eq. [T] when the 
experimental result Xi uniquely specifies a corresponding 
state of knowledge yj. In this case p{yj\xi) — 1, and 
Eq. [l] quantifies our surprise at finding yj to be correct. 

Since the information gain is the appropriate measure 
of scientific merit of an experimental result Xi, the scien- 
tific merit of a proposed experiment should be quantified 
by the information that would be gained from each pos- 
sible experimental outcome, weighted by the expectation 
p{xi) that each experimental outcome Xi is obtained. The 
information content AH in an experiment, defined ear- 
lier in Eq. [6] can be shown with a few lines of algebra [37] 
to be equal to its expected information gain, 

AH ~ 'y^^p{xi) information gain (xi), (9) 

i 

as required for logical consistency of the concepts that 
have been introduced. 

This section has introduced an additional level of so- 
phistication beyond Sec. |II A| The elementary notion of 
surprisal in Eq. [l] quantifying the (a posteriori) scien- 
tific merit of an experimental result, is replaced by the 
information gain of Eq. [Sj The elementary notion of in- 
formation content in Eq. [2] quantifying the (a priori) 
scientific merit of a proposed experiment, is replaced by 
Eq.§ 
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C. Synopsis 

The essential thesis of this article is summarized in two 
sentences. 

• The appropriate quantification of scientific merit of 
a proposed experiment or analysis (before it is per- 
formed and its outcome is known) is the reduction 
in information entropy the experiment or analysis 
is expected to provide, defined by Eq. |6] 

• The appropriate quantification of scientific merit of 
an experiment or analysis after the result is known 
is the information gained from the result, defined 
by Eq. |8] 

III. EXAMPLES 

The specific figure of merit defined above can be used 
to quantify the scientific merit of any experiment or anal- 
ysis. This section illustrates the application of this figure 
of merit to particular experiments and analyses in the 
field of particle physics. 

In these examples it will be clear that a priori esti- 
mates of the probability of possible outcomes must be 
made. No prescription exists for choosing these a priori 
probabilities of possible outcomes. The examples below 
will nonetheless demonstrate that (1) the same scientific 
conclusion is often reached for the full range of justifi- 
able a priori probabilities, and (2) the formulation of the 
discussion in terms of the assignment of these a priori 
probabilities is frequently productive and enlightening. 
The fact that there is not a well-developed literature to 
point to for the justification of these a priori probabilities 
emphasizes the fact that up to this point the importance 
of these probabilities has not been properly recognized in 
assessing the scientific merit of proposed and completed 
experiments. 

A. Proposed experiments 

For the purposes of the calculations below, the pos- 
sibility of new physics being found at Tevatron Run II 
is taken to be 20%, with 10% in each of Run Ila and 
Run lib, and with equal probability to be found in any 
of roughly 10^ potentially interesting kinematic distribu- 
tions. A more detailed justification of these numbers is 
provided in Appendix |A 1[ An observed discrepancy is 
assumed to contain sufficiently detailed information to 
reduce the space of qualitatively different possible expla- 
nations for the discrepancy by a factor of 10^. These 
numbers will naturally be the subject of significant and 
productive debate in any real program review. One of the 
purposes of the calculations below is to provide intuition 
for the way in which conclusions change with different 
probability assignments, and the extent to which they 
do not. 



• Flipping a coin has two possible outcomes. Let 
xi denote the experimental outcome in which the 
coin comes up heads, and let X2 denote the ex- 
perimental outcome in which the coin comes up 
tails. Let yi denote the state of knowledge in 
which the Standard Model Higgs boson is known 
to exist, and let ?/2 denote the state of knowl- 
edge in which the Standard Model Higgs boson 
is known not to exist. Whether the coin comes 
up heads or tails tells us absolutely nothing about 
whether the Standard Model Higgs boson exists 
or not; hence p{vi\xi) = p{yi\x2) = p{yi), and 
p{y2\xi) = p{y2\x2) = p{V2)- In words, our de- 
gree of belief p{yi\xi) that the Higgs boson ex- 
ists given that the coin comes up heads is equal 
to our degree of belief p{y i\x2) that the Higgs bo- 
son exists given that the coin comes up tails, which 
in turn is equal to our degree of belief p{yi) that 
the Higgs boson exists before performing the ex- 
periment. In this case from Eq. |5] the expected 
information entropy after performing the experi- 
ment H{Y\x) = -J2t,jPl^i^yj)^og^oPiyj\^i) = 

— J2j PiVj) ^'^SioPiVj) = ^0^)i the information en- 
tropy before performing the experiment, and the 
scientific merit of this experiment is AH = H{Y) — 
H{Y\X) = 0. 

This example illustrates that AH quantifies scien- 
tific merit (and not merely the number of possible 
experimental outcomes) through its treatment of 
qualitatively different states of scientific knowledge 
Y. Experiments with varied possible outcomes X 
that do not inform the probabilities we assign to 
the qualitative states of knowledge Y have zero sci- 
entific merit. 

• The /i2e experiment [21] has two possible outcomes. 
Appendix |A 2| argues that an excess of muon to elec- 
tron conversions will be observed with an expecta- 
tion of « 18%. With probability of 1 - 0.18 = 0.82 
results consistent with the Standard Model will 
be obtained. In either case the experiment re- 
sults in a state of knowledge with zero informa- 
tion entropy, so H{Y\X) = 0. The information 
content in this analysis is thus AH = H{Y) — 
H{Y\X) = H{Y)~0 = -(0.18xlogio(0.18)+0.82x 
logio(0.82)) w 2e-01 

The most important factor for determining the sci- 
entific merit of this analysis is the estimated prob- 
ability of 18% of observing an excess of muon to 
electron conversions, which sets the scale for the 
computed scientific merit. Note that for an experi- 
ment such as fi2e with binary outcome the scientific 
merit is maximal when the estimated probability 
for a signal is 50%. 

• A search for single top has two possible outcomes. 
As argued in Appendix |A 1[ with probability 1 — 
10~^ = 0.99999 single top will be found as pre- 
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dieted by the Standard Model, treating this anal- 
ysis as just one of 10'' distributions that can be 
considered in Tevatron Run II. With probability 
10~^ single top will be found to not be consistent 
with the Standard Model prediction. The expected 
reduction in information entropy from this anal- 
ysis is thus Ai/ = H{Y) - H{Y\X) = H{Y) - 







(10-5 X logio(10-5) + {1- 10-5)logio(l 



10-5)) « 5e-05. 

The dominant factor in determining the scientific 
merit of this analysis is seen to be the estimated 
probability of not observing single top as predicted 
by the Standard Model. In this example, as in all 
others, the possibility of an outright experimental 
mistake is ignored. 

• A search for a gluino (g) has ~ 101 possible out- 
comes. The probability for new physics to be found 
in Tevatron Run Ila is 10%, with this probabil- 
ity divided equally among 10* distributions. The 
probability for new physics to be found in the par- 
ticular distribution used in the gluino analysis is 
thus 10%/10* — 10^^. If new physics is found, the 
signal events will contain information that should 
narrow down the qualitatively different possible ex- 
planations for a discrepancy in this distribution by 
a factor of 10^. That is, for every 100 qualita- 
tively different plausible interpretations (Bard sto- 
ries [5]) that could be proposed for seeing a discrep- 
ancy in the particular kinematic distribution con- 
sidered, the characteristics of the observed events 
are expected to allow the elimination of 99 of them. 
The probability that no new physics is found in 
this analysis is 1 — 10^^ There are thus 10^ possi- 
ble outcomes with probability 10-^ corresponding 
to 10^ possible different states of understanding in 
the event of a discovery, and one possible outcome 
with probability 1 — 10-^ = 0.99999 corresponding 
to no discovery. The information content in this 
analysis is AH = H{Y) - H{Y\X) = H{Y) - = 
-(10^ X 10-7 X logio(10-7) + (1 - 10-5) X logio(l - 
10-5)) « 7e-05. 

The dominant factor in determining the scientific 
merit of this analysis is seen to be the estimated 
probability for discovery, which sets the scale of 
10-5, adjusted slightly by the decimal logarithm of 
the estimated power of the analysis for discriminat- 
ing among competing interpretations in the event of 
a discovery. This analysis can be taken as a proxy 
for any similar dedicated Tevatron search. 

• A search for the Standard Model Higgs boson at 
the Tevatron has (qualitatively) three possibilities. 
The Standard Model Higgs boson exists with prob- 
abihty 95%, and does not exist with probability 5%. 
If the Standard Model Higgs boson exists, it will be 
seen at the Tevatron with probability 10%. Possi- 
bilities are that the Standard Model Higgs boson 



exists and is seen at the Tevatron, with probabil- 
ity ^(exists, observed) = 95% x 10% = 0.095; that 
the Higgs boson exists but is not seen at the Teva- 
tron, with probability ^(exists, [observed) — 95% x 
90% — 0.855; and that the Higgs boson does not ex- 
ist and is not seen at the Tevatron, with probability 
p(!exists, [observed) = 5%. The entropy of the state 
of knowledge before the experiment is performed is 
H{Y) = //(exists?) = -(0.95 x logio 0-95 + 0.05 x 
logio0.05) « 0.086 [40]. Unlike the previous ex- 
amples, in which the experiment results in an in- 
formation state with zero entropy, in the search 
for the Higgs boson at the Tevatron we may be 
left in an information state with non-zero entropy. 
The scientific merit is the expected reduction in 
entropy AH = H{Y) - H{Y\X) = //(exists?) - 
//(exists? [observed?) that will be provided by 
the experiment, where //(exists? [observed?) = 
— (p(exists, observed) x logiQp(exists|observed) + 
^(exists, [observed) x logiQp(exists|!observed) + 
p(!exists, [observed) x logjQp(!exists|!observed)) « 
-(0.095 X logio(l) -I- 0.855 x logio(0.945) + 0.05 x 
logio(0.055)) « 0.084. The scientific merit is thus 
calculated to be AH = H{Y) - H{Y\X) « 0.086 - 
0.084 = 2e-03. 

The important factors for determining the scientific 
merit of this analysis are the estimated probability 
of 95% of observing the Higgs, which sets the scale 
of a few xlO-^, and the estimated probability of 
10% that the experiment will provide an unambigu- 
ous answer, which reduces the scientific merit to a 
few xlO-^. 

• Rewinding back to the year 2003, the Tevatron 
Bs mixing measurement with 1 fb-^ of integrated 
luminosity similarly has (qualitatively) three 
possibilities. With initial results from the B fac- 
tories already in hand, the Standard Model CKM 
picture will be insufficient to accommodate the 
Tevatron measurement of A m^ w ith probability 
10-5, as argued in Appendix Al The projected 
sensitivity of the CDF Bg mixing analysis with 
1 fb-Ms Aras = 30 ps-^ The Standard Model 
can accommodate a value of Arus up to and 
including 30 ps-^ at 95% confidence level. If 
the CKM picture is sufficient, with probability 
95% a measurement of Am^ will made with 
Arris < 30 ps-^, confirming that the Standard 
Model CKM picture is indeed sufficient. Pos- 
sibilities are that the CKM picture is sufficient 
and Bs mixing will be observed, with probability 
p(ckm, observed) ^ 0.99999 x 95% w 0.94999; 
that the CKM picture is sufficient but 
Atos is sufficiently large that Bs mixing is 
not seen at the Tevatron, with probability 
p(ckm, [observed) = 0.99999 x 5% « 0.0499995; 
and that the CKM picture is insufficient 
and Bs mixing is not seen at the Tevatron, 



with probability p(!ckm, lobserved) = 10~^. 
The entropy of the state of knowledge 
before the experiment is performed is 
H{Y) = i7(ckm?) = -(0.99999 x log^ 0.99999 + 
0.00001 X logio 0.00001) « 5 X 10"^ [41 . As in the 
previous Higgs example, this experiment may re- 
sult in an information state with non-zero entropy. 
The appropriate figure of merit is the entropy 
reduction AH = H{Y) - H{Y\X) = iJ(ckm?) - 
i?(ckm? [observed?) , where i/(ckm? [observed?) = 
— (p(ckm, observed) x logjQp(ckm|observed) + 
p(ckm, [observed) x logiQp(ckm| [observed) + 
p(!ckm, [observed) x log]^oP('ckm| [observed)) « 
-(0.94999 X logio(l) + 0.0499995 x logio(0.9998) -f 
10~^ X logio(0.0002)) w 4 X 10"^ The scientific 
merit is the reduction in information entropy 
expected to be provided by the experiment, 
calculated to be AH = H{Y) - H{Y\X) w 
5 X 10~5 - 4 X 10"^ = le-05. 

The most important factor for determining the sci- 
entific merit of this analysis is the estimated prob- 
ability of 10~^ of not observing Bs mixing, which 
sets the scale of a few xlO^^. Unfortunately the 
sensitivity of this analysis is such that it is unable 
to cash in on the (sur)prize of providing conclu- 
sive evidence that the CKM picture is insufficient. 
It is instructive to see how this limited sensitivity 
(resulting in the number of 5% in the calculation 
above) propagates through the calculation into a 
relatively large value for H{Y\X), resulting in an 
expected information entropy H{Y\X) = 4 x 10^^ 
after the measurement that is comparable to the 
information entropy H{Y) = 5 x 10^^ before the 
measurement, leaving an expected information gain 
of only AH = lx 10"^ 

A global analysis of all Tevatron Run II data will 
discover nothing with a probability of 80%. With 
probability 20% a discovery will be made in one of 
10'* distributions considered, with 10^ qualitatively 
different possible states of understanding resulting 
from a discrepancy observed in any of these dis- 
tributions. In the event of a discovery, there are 
thus 10^ possible states of understanding, each with 
probability 2 x 10^^. The information content in 
this analysis is thus AH = H{Y)-0 = -(10^ x (2x 
10-7) xlogio(2x 10^7)-!- 80% xlogio(80%)) = 1.4. 

The dominant factors influencing the scientific 
merit of this analysis are the estimated probability 
of 20% of discovering new physics at Tevatron Run 
II, which sets the scale of 0.2, and the decimal log- 
arithm of the power of the analysis to discriminate 
among possible interpretations, which provides an 
additional factor of seven. 

Integrating over all analyses that will be performed 
at the CERN Large Hadron Collider in the next five 
years, a significant discovery will be made with a 



Analysis Merit Cost Bang per buck 
(M$) (Merit per M$) 



LHC, all searches 
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5e-H03 


le- 


-03 


Tevatron II global search 


1.4 


3e-01 


5 




^2e 


2e- 


■01 


le-H02 


2e- 


-03 


Tevatron Higgs search 


2e- 


■03 


le-HQl 


2e- 


-04 


Bs mixing 


le- 


■05 


le+01 


le- 


-06 


g search 


7e- 


■05 


le-01 


7e- 


-04 


single top search 


5e- 


■05 


5 


le- 


-05 



flipping a coin le-07 

TABLE I: The scientific merit of selected experiments and 
analyses, calculated in terms of expected information content 
AH with decimal logarithm as described in the text, and 
ordered according to decreasing merit. Estimated incremental 
cost for each experiment or analysis is provided in units of 
millions of U.S. dollars. Scientific "bang per buck" is obtained 
by dividing the calculated merit by the estimated incremental 
cost. 

probability of 90%. The space of new physics mod- 
els is so large that there are ^ 10* qualitatively dis- 
tinct states of possible understanding accessible in 
the event of a discovery. With probability of 10% a 
Standard Model Higgs boson will be seen and noth- 
ing else. The information content in the entire LHC 
is thus AH = H{Y) - = -(10** x (90% x 10"*) x 
logio(90% X 10-8) + 10% X logio(10%)) = 7.4. 

In this case the 0(1) discovery potential sets the 
scale for LHC's scientific merit, and the power of 
the direct production of new states to distinguish 
among competing interpretations provides an addi- 
tional factor of seven. 

The assignment of a priori probabilities above is cer- 
tainly open for debate, and the reader may object that 
the problem of quantifying an experiment's scientific 
merit has simply been reformulated in terms of the esti- 
mation of the probabilities of possible experimental out- 
comes. At worst, this reformulation significantly changes 
and focuses the discussion. In most cases, the scien- 
tific conclusion drawn from this reformulation is robust 
against the variation of these estimated probabilities 
within their justifiable range. In cases where the scien- 
tific conclusion is not robust against such variation, this 
framework points out precisely what factors are impor- 
tant. The fact that there is not a well-developed liter- 
ature to point to for the justification of the above esti- 
mated probabilities emphasizes the fact that until now 
these estimated probabilities have not been regarded as 
the important ingredients in assessing the scientific merit 
of proposed and completed experiments. 

The scientific merit of the proposed experiments and 
analyses considered above can in this way be quantified 
and ordered, as in Table |l] Also included in this table 
is an estimated incremental cost for performing the ex- 
periment or analysis, which is subject as always to some 
ambiguity in the choice of accounting [42 . Dividing the 
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scientific merit by the estimated cost provides a "scien- 
tific bang per buck," in units of information content per 
million dollars. Since information entropy has the de- 
sired behavior under addition (in the sense that if two 
experiments together provide the same information as a 
third experiment, the sum of the merits of the first two 
experiments equals the merit of the third), the scientific 
merit per unit cost is appropriate for judging the most 
cost effective allocation of limited resources, and for com- 
paring experiments of significantly different scales. This 
"scientific bang per buck" provides a quantitative frame- 
work for advocates of experiments of all sizes to argue 
their cases on even footing, and for review panels to as- 
sess such cases on even footing. 

In this context, the value of making data publicly avail- 
able (such as through Quaero [6j|7j) is to reduce the cost 
of dedicated analysis (such as the g search in Table |l| 
from 100 k$ (corresponding to two years of a graduate 
student's time) to 100 $ (corresponding to two hours of 
a graduate student's time), increasing the scientific bang 
per buck for such a search by three orders of magnitude. 

Table [l] is particularly interesting and potentially valu- 
able to the extent that its conclusions are at odds with 
accepted wisdom of the relative scientific merit of the 
experiments and analyses listed. If indeed the figure of 
merit proposed here is a suitable measure of the scien- 
tific merit of an experiment or analysis, then accepted 
wisdom can be dangerously misleading. If accepted wis- 
dom is correct, then there should exist an alternative and 
justifiably more appropriate quantifiable figure of merit 
than the one proposed in this article. If the latter, this 
article should help motivate its development. 



B. Completed experiments 

The figure of merit suggested for assessing the scien- 
tific merit of a completed experiment is the information 
gained from the result obtained, quantified by Eq.[8] This 
section illustrates the application of this figure of merit to 
historical, recent, and hypothetical future experimental 
results in our field. 

In these examples, as in those in the previous section, 
it will be clear that estimates of the probability of possi- 
ble outcomes must be made. For completed experiments, 
like for future experiments, the relevant probabilities are 
those that quantify our expectation of probable outcomes 
before the execution of the experiment. Here as before, 
no prescription exists for choosing these a priori prob- 
abilities of possible outcomes. The examples below will 
nonetheless demonstrate again that (1) the same scien- 
tific conclusion is often reached for the full range of jus- 
tifiable a priori probabilities, and (2) the formulation of 
the discussion in terms of the assignment of these a pri- 
ori probabilities is frequently productive and enlighten- 
ing. Provided historical references, although not explic- 
itly quantifying expectation, nonetheless give a fiavor of 
the rhetoric of the time that can be used to justify rea- 



sonable a priori probabilities. 

• Returning to the example of the flipped coin, let 
xi denote the experimental outcome in which the 
coin comes up heads, X2 the experimental outcome 
in which the coin comes up tails, yi the state of 
knowledge in which the Standard Model Higgs bo- 
son is known to exist, and 1/2 the state of knowledge 
in which the Standard Model Higgs boson is known 
not to exist. As before, p{yi\xi) = p{yi\x2) = 
p{yi), and p{y2\xi) = p{y2\x2) = pivi)- Assume 
the coin comes up heads: the experimental out- 
come xi is realized. 

In this case from Eq. [8] the information gain 
of this outcome is - J^jPiVjl^i) logio pfa^i) = 

-E,P(2/.)logiog|i = -E,P(2/.)logiol = 0. 
The scientific merit of this experimental outcome, 
quantified by its information gain, is 0. 

• Before the W and Z bosons were discovered [HI IHJ 
[TUl [TT| . these electroweak gauge bosons were ex- 
pected to exist with a degree of confidence suffi- 
ciently high that the SppS was specifically designed 
and built for their discovery [T^]. In the late 1970's, 
the W and Z bosons were assumed to exist with a 
probability of 95% [13 . The information gain of 
the discovery of these gauge bosons in 1983 was 
thus -logio(95%) « 0.02. 

• Before the top quark was discovered at Fermilab 
in 1995 [2] [13], it was assumed to exist with 
a degree of confidence sufficiently high that the 
Tevatron to a great extent was specifically de- 
signed and built for this discovery [16 . In the late 
1980's, the top quark was assumed to exist with a 
probability of 95% [17 . The information gain of 
the discovery of the top quark in 1995 was thus 
-logio(95%) « 0.02. 

• In 1990, it was believed with a probability of 30% 
that new electroweak scale physics (such as super- 
symmetry) would appear in the running of Teva- 
tron Run I or LEP 2. The suprisal that no new 
physics was discovered at Tevatron Run I or LEP 2 
was thus -logio(l - 30%) » 0.15 [43). 

• The discovery of the J/^ [TSJ [TS] was largely unan- 
ticipated; in 1973 few professional physicists took 
the notion of physical quarks seriously [SIT. The 
expectation that experiments at Brookhavcn and 
SLAC would produce evidence of a narrow reso- 
nance confirming the physicality of the quark pic- 
ture was small, corresponding to a probability of 
~ 10^^. The information gain of the discov- 
ery of the J/^f in November of 1974 was thus 
-logi„(10-2) = 2. 

• The discovery of the t lepton [HI [22l [S^ was also 
quite unanticipated; in the early 1970's experimen- 
tal results fit nicely into a framework with only two 



generations of fermions. Indeed, there was less the- 
oretical motivation for expecting the r lepton than 
expecting to see the J/^ [53]. The expectation that 
a third generation would be discovered thus corre- 
sponded to a probability even less than the number 
used above for the J/^* discovery. If the a priori 
probability of a SLAC experiment producing evi- 
dence for a r lepton in 1977 is taken to be 10"'^, 
then the information gain of this experimental re- 
sult is -logio(10"^) = 3. 

By 1977, the existence of a third generation of lep- 
tons was established [23) . The expectation within 
the community at this time that a h quark would 
be observed can be estimated to correspond to a 
probability of ~ 30% [25]. The discovery of the 
T [26 thus represents an experimental result with 
scientific merit equal to — log]^Q(30%) « 0.5. 

The determination of Am^ has recently been im- 
proved at the Tevatron with 1 fb~^ of integrated 
luminosity through an analysis of mixing |27j . 
Before the observation of Bs mixing, the CKM pic- 
ture was expected to provide an inadequate de- 
scription of this phenomenon with a probability 
of 10~^. Quantified in terms of information gain, 
the scientific merit of this experimental result is 
-login(0. 99999) = 4e-06. 

A search for a gluino {g) has been performed at the 
Tevatron, with no evidence of new physics found. 
From the previous section, the probability of find- 
ing a gluino in this search was lO^'^. The scien- 
tific merit of the result of this search, in which 
no evidence of physics beyond the Standard Model 
is found, is quantified by a information gain of 
-logio(0. 99999) = 4e-06. 

A global analysis of CDF Run Ila high-p^ data has 
recently been performed [2EII1S!, and no evidence of 
new physics has been found. Using numbers from 
the previous section, the probability that no evi- 
dence of new physics would be found in this anal- 
ysis was 90%. The scientific merit of this result is 
thus equal to -logio(90%) = 0.05. 

Fast-forwarding in time, consider the scientific 
merit of a null result obtained in a global search 
of Tevatron Run lib high-pj^ data. Using numbers 
from the previous section, the probability that no 
evidence of new physics would be found in these 
data is 90%. The scientific merit of such a result 
would thus be equal to — log]^Q(90%) = . 05. 

The total scientific merit of a null result being ob- 
tained in global searches at Tevatron Run Ila and 
Tevatron Run lib would be 0.05 + 0.05 = 0. 10. 

Fast-forwarding in time, consider the scientific 
merit of the discovery of a Standard Model Higgs 
boson. The overwhelming expectation of the field 



Result Merit Cost Bang per buck 

(M$) (Merit per M$) 



r discovery 


3 




6e- 


■01 


5e-H00 


J/*!/ discovery 


2 




le+01 


2e- 


-01 


there is no Higgsf 


l.c 


3 


5e+03 


3e^ 


-04 


T discovery 


5e- 


-01 


1 




5e^ 


-01 


null Tevatron I + LEP 2 


2e- 


-01 


3e+03 


6e^ 


-05 


global null Tevatron Ila 


5e- 


-02 


3e- 


■01 


2e- 


-01 


global null Tevatron Ilbf 


5e- 


-02 


3e- 


■01 


2e- 


-01 


W and Z discoveries 


2e- 


-02 


5e+02 


Ae- 


-05 


top quark discovery 


2e- 


-02 


5e+01 


Ae- 


-04 


Higgs discoveryf 


2e- 


-02 


5e+03 


Ae- 


-06 


Bs mixing observation 


4e- 


-06 


le+01 


Ab- 


-07 


g search 


4e- 


-06 


le- 


■01 


Ae- 


-05 


single top discovery! 


4e- 


-06 


5 




Ae- 


-06 


coin comes up heads 







le- 


■07 








TABLE II: Table of the scientific merit of selected historical, 
recent, and hypothetical future (f) experimental results, cal- 
culated in terms of information gain with decimal logarithm 
as described in the text, and ordered according to decreas- 
ing merit. Estimated incremental cost for each experiment or 
analysis is provided in units of millions of U.S. dollars. Scien- 
tific "bang per buck" is obtained by dividing the calculated 
merit by the estimated incremental cost. 

is that a Standard Model Higgs boson exists, with 
a probability taken to be ~ 95%. The informa- 
tion gain of obtaining this experimental result is 
-logio(95%) = 0.02. If the LHC instead rules 
out the possibility of the existence of a Standard 
Model like Higgs boson, the scientific merit of such 
a result would be significantly greater, with infor- 
mation gain equal to — log]^Q(5%) =1.3. 

• Fast-forwarding in time, consider the scientific 
merit of the discovery of Standard Model single 
top production. The overwhelming expectation is 
that the top quark is produced singly in pp colli- 
sions as predicted by the Standard Model. Taking 
the probability that Nature does not singly produce 
top quarks from the previous section to be 10~^, as 
argued in Appendix ] A 1| the scientific merit of the 
discovery of single top production is quantified by 
a information gain of — log]^Q(0. 99999) = 4e-06. 

The assignment of probabilities of possible outcomes 
above is certainly open for debate. The claim is made 
that the physics discussion of these probabilities forms a 
particularly focused, fruitful, and constructive framework 
for assessing the scientific merit of experimental results. 

The scientific merit of the experimental results con- 
sidered above can thus be quantified and ordered, as in 
Table [Hi Also included in this table is an estimated in- 
cremental cost for performing the experiment or analy- 
sis, which is subject as always to some ambiguity in the 
choice of accounting. Dividing the scientific merit by the 
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estimated cost provides a "scientific bang per buck," in 
units of information content per million dollars. 

Table [TT| is particularly interesting and potentially valu- 
able to the extent that its conclusions are at odds with 
accepted wisdom of the relative scientific merit of the ex- 
periments and analyses listed. Note that the scientific 
merit, quantified in terms of the information gain, obeys 
the desired property of addition, such that if two exper- 
iments together come to the same conclusion as a third 
experiment, then the sum of the scientific merits of the 
first two experimental results is equal to the scientific 
merit of the third result. The scientific merit of the dis- 
covery of the r lepton is seen to equal the total scientific 
merit of all other present and past experimental results 
listed in Table a conclusion that is robust against rea- 
sonable variation in estimated prior probabilities. The 
scientific merit of a Tevatron global search is seen to be 
10'' times that of a dedicated, model-dependent Tevatron 
search, consistent with the relative scope of data space 
analyzed in each case. The scientific merit of the com- 
bined null result of Tevatron I and LEP 2 is measured 
to be twice the combined scientific merit of the discov- 
ery of the W boson, the discovery of the Z boson, the 
discovery of the top quark, and a future Higgs boson dis- 
covery. The scientific merit of not finding a Standard 
Model Higgs boson at the LHC is nearly two orders of 
magnitude larger than the scientific merit of discovering 
the Standard Model Higgs boson. The scientific mer- 
its of the "discoveries" of the W boson, Z boson, top 
quark. Standard Model Higgs, Bg mixing, and single top 
production range from two to five orders magnitude less 
than the scientific merits of the "null results" that would 
have corresponded to a non-observation. 



IV. SUMMARY 

The choice of a reasonable quantitative figure of merit 
for assessing the scientific merit of proposed experiments 
can inform and focus program review and accompany- 
ing decisions of resource allocation in many subfields of 
the physical sciences. The related choice of a reasonable 
figure of merit for assessing the scientific merit of any 
particular experimental result would inform the evalu- 
ation of those organizations and individuals responsible 
for the production of the result. 

This article advocates that the amount of information 
an experiment is expected to provide is the appropriate 
figure of merit for assessing the scientific merit of any 
experiment whose result is not yet known. The amount 
of information an experiment is expected to provide is a 
concept with a well developed theory that is nearly sixty 
years old, quantified by a change in information entropy 
AH in the context of information theory. 

This article advocates that the extent to which an ex- 
perimental result surprises provides the appropriate fig- 
ure of merit for assessing the scientific merit of any spe- 
cific experimental result. The extent to which an experi- 



mental result surprises is also an elementary notion in the 
context of information theory, quantified by the result's 
information gain. 

Use of information content or information gain to eval- 
uate the scientific merit of experiments requires the esti- 
mation of the probabilities of qualitatively different out- 
comes, and the reader may object that the problem of 
quantifying an experiment's scientific merit has simply 
been reformulated in terms of the estimation of the prob- 
abilities of possible experimental outcomes. At worst, 
this reformulation significantly changes and focuses the 
discussion. The fact that there is not a well-developed 
literature to point to for the justification of these a priori 
probabilities emphasizes the fact that until now the im- 
portance of these probabilities has not been properly rec- 
ognized in assessing the scientific merit of proposed and 
completed experiments. In most cases, the scientific con- 
clusion drawn from this reformulation is robust against 
the variation of these estimated probabilities within their 
justifiable range. From Tables [I| and [III it is seen that the 
scientific merit per incremental unit cost ranges over five 
orders of magnitude for the representative experiments 
and analyses considered. 

The reader may object to the very idea of constructing 
an explicit figure of merit to quantify the scientific merit 
of experiments. Such a reader misses the point that this 
is done (implicitly, if not explicitly) every time a deci- 
sion of resource allocation is made. It is surely in the 
field's best interest for such evaluations to be made in 
the sharpest, most open, most quantifiable, and scientif- 
ically best motivated framework possible. 

The reader may object to the specific figure of merit 
advocated in this article, particularly if his or her exper- 
iment scores poorly. Such a reader is challenged to find 
a scientifically better motivated figure of merit. 

Presumably nearly all readers will take issue with a 
few of the numbers in the examples. Readers should use 
these to assess fo r th emselves the claims made at the 
beginning of Sec. |III[ (1) the same scientific conclusion 
is often reached for the full range of justifiable a priori 
probabilities, and (2) the formulation of the discussion in 
terms of the assignment of these a priori probabilities is 
frequently productive and enlightening. 

In the context of the scientific merit advocated in this 
article, the point of making frontier energy collider data 
publicly available (such as with QuAERO [6l|7]) is to re- 
duce the cost of an analysis like the g search in Tables |l] 
and [n] from ^$100,000 to ^$100, increasing the scien- 
tific bang per buck of such an analysis by three orders of 
magnitude. 

The figure of merit described here may provide a 
novel and useful quantitative framework within which 
(HEPAP, P5, NSF, and DoE) decisions about future re- 
source allocation can be made. 
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APPENDIX A: DERIVATION OF EXPECTATION 

This appendix provides arguments for the numbers 
used to quantify expectation in the main text of this ar- 
ticle. The quantitative measure introduced in this article 
is sufficiently novel that the field so far has little practice 
with the type of argument required; reference to existing 
literature is in most cases insufficient. The quantitative 
arguments constructed below are almost certainly also in- 
sufficient in some respects, but are provided as a starting 
point for further improvement and discussion. 



1. LEP, Tevatron, and LHC 

This section argues the discovery probabilities of LEP, 
the Tevatron, and the Large Hadron Collider. A prior 
is constructed to represent our expectation in 1990 that 
the first sign of physics beyond the Standard Model will 
appear at or slightly above the electroweak scale. This 
prior is then evolved through the experimental realities 
of Tevatron Run I and LEP 2 to update expectation to 
the year 2000. This expectation is then evolved through 
the experimental reality of Tevatron Run Ila and a hy- 
pothetical null result in Tevatron Run lib to determine 
what fraction of expectation from 2000 is left at the end 
of Tevatron running in 2010. The fraction of expecta- 
tion from 1990 that has been eliminated by 2000, 2010, 
and 2020 (corresponding to the discovery probabilities of 
LEP, the Tevatron, and the LHC) is determined. 

Prior probabilities can be defined given the field's 
knowledge in 1990, before Tevatron Run I (1990-1996) 
or LEP 2 (1996-2000). The production of a new process 
beyond the Standard Model is largely characterized by 
an overall coupling strength g and the mass m of the 
new state that is produced. Our knowledge circa 1990 
is thus characterized by a prior p{g, m|1990) for the cou- 
pling g and the mass m of the most accessible non Stan- 
dard Model physics process. This prior is assumed to 
factorize, such that 



p(g,m|1990) = p(g|1990)p(m|1990). 



(Al) 



The prior for the coupling p(g|1990) is taken to be 
uniform between and 1.5, noting that for weak pro- 



cesses g = ^/47raEM ~ 0.3 and that for strong processes 
g = ^47ras w 1.2. Exphcitly, 



p(5|1990) = (0 <g<1.5)/1.5, 



(A2) 



where (0 < 5 < 1.5) evaluates to 1 if true, and if false. 
The prior p(m| 1990) for the mass m of the most accessible 
new state is taken to be 



p(m|1990) 



A2 



exp (— rn/A), 



(A3) 



where A = 150 GeV sets the peak of the distribution 
p(m|1990), as shown in Fig.[l|a). This prior reflects the 
prejudice that new physics should appear at or slightly 
above the mass of the W and Z bosons. The joint prior 
p((7,m|1990) is shown in Fig.[ljb). 

The proton and antiproton parton distrubtion func- 
tions are such that the single particle production cross 
section (pp X) falls by a factor of two for every in- 
crease of TO by Xi/2 — 40 GeV, holding g constant. By 
the same argument, the pair production cross section 
{pp XX) falls by a factor of two for every increase 
of m by 20 GeV, holding g constant. Letting n — 1 de- 
note single particle production and n = 2 denote pair 
production, the number of signal events s varies with g, 
TO, n, and the integrated luminosity L according to 



(A4) 



The number of background events b varies only with the 
integrated luminosity, so that b oc L. Combining these, 
the figure of merit is 



Vb 



Lg^ 2^'"^^/ ^1-/2 



(A5) 



In Tevatron Run I, the top quark was discovered at 
the edge of experimental sensitivity. This discovery cor- 
responded to the strong coupling g = 1.2, a mass to equal 
to the top quark mass rrit = 175 GeV, pair production 
of rt = 2, and an integrated luminosity of L « 100 pb^^. 
Inserting these numbers into Eq. |A4| yields 



Vb 



100 pb"^ (1.2)^2 



-2mt/Ai/2 



(A6) 



top 



Taking the top quark discovery as a relevant benchmark, 
a new process characterized by g, to, and n will be ob- 
served at the Tevatron after collecting an integrated lu- 
minosity L if 



L 



2 2-^"^/^i/2 



100 pb"^ V1.2/ 2-2™tMi/2 
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> 1, 



where this expression is obtained by dividing Eq 
Eq 



(A7) 



5] by 



A6 



Equation A7 implies that given the existence of 
a new physics process characterized by a coupling g, a 
mass scale to, and single or pair production n, the prob- 
ability of observing a null result at the Tevatron upon 
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p(g,m|1990) p(g,m|1996) 




FIG. 1: (a) The distribution p(mjl990), representing 1990 expectation for the mass of the next non Standard Model particle to 
be discovered. The scale of all mass axes is in units of GeV, and all distributions extend past the 500 GeV upper threshold used 
in constructing these plots, (b) The distribution p{g,m\1990), representing the joint prior expectation for the coupling g and 
mass m characterizing the next non Standard Model particle to be discovered, (c) The distribution p{g, m|1996), reflecting our 
prior from 1990 with the understanding gained from the null result of Tevatron Run I. The effect of Tevatron Run I is to "zero" 
a substantial piece of the 1990 prior. What is left is renormalized to unit integral. This and subsequent plots show the case of 
pair production, with n — 2. The shape of the region zeroed by Tevatron Run I is determined considering the top quark as a 
representative benchmark for the sensitivity of Tevatron Run I in the variable space of coupling g and mass m, as described 
in the text, (d) The distribution p{g , m\2000) , reflecting the previous distribution with the additional knowledge of the null 
result from LEP 2. The effect of LEP 2 is to zero that part of the density with m < 100 GeV. (e) The distribution p{g, m|2006), 
reflecting the previous distribution with the additional knowledge of the null result of Tevatron Run Ila. The effect of Tevatron 
Run Ila is to zero ~ 10% of the density in the previous distribution, (f) The distribution p{g, m\2010), reflecting the previous 
distribution assuming a null result is obtained in Tevatron Run lib. The effect of Tevatron Run lib is to zero ~ 12% of the 
density in the previous distribution. All distributions (a)-(f) integrate to unity. As a sanity check, note that the Standard 
Model Higgs {g ^ 0.3,m « 130) and single top production {g « 0.3,m ^ 175) are at the mode of }3(gi, m|2006). 



the accumulation of integrated luminosity equal to L is I, from application of Bayes's theorem 



p(tev(L)|1990,5,m) = \ g< 



1.2 X 2("'"/2^™*)/-''i/2 



( L \ 

\ 100 pb-1 J 



1/4 



(A8) 



where the right hand side is just a rewriting of Eq. |A7| 
using the notation that the boolean expression evaluates 
to 1 if true, and if false. 

The expectation in 1990 that new physics would be 
seen at Tevatron Run I or LEP 2 is determined to be 



1 -p(tevl,lep2|1990) = 30%. 



(A9) 



Letting p{g, m\1996) denote our knowledge in 1996, 
comprising the prior from 1990 plus the additional knowl- 
edge (tevl = tev(100 pb"^)) of no processes beyond the 
Standard Model having been observed in Tevatron Run 



p{g, m\1996) 



p(5,m|1990,tevl) 
39(5, m, tevl 1 1990) 

p(tevl|1990) 
p(tevl|1990, g, m)p{g, m|1990) 



35(tevl|1990) 



(AlO) 



Here p(tevl|1990,5, m) is defined by Eq. A8 and 



p(tevl|1990) quantifies the probability of having obtained 
the null Tevatron Run I result given our knowledge in 
1990, representing the density left in going from Fig.[l|b) 
to Fig. ^c), which shows a plot of p{g, m|1996). 

Summarizing the null result from LEP 2 as ruling out 
the possibility that m/n < 200 GeV leads to 



p(lep2|1996,. g,m) = (— > 200 GeV), 



(All) 



using again the notation that the boolean expression eval- 
uates to 1 if true, and if false. Letting ^(g, to|2000) 
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denote our understanding in 1996 with the additional 
knowledge (lep2) of no processes beyond the Standard 
Model having been observed in LEP 2, a similar applica- 
tion of Bayes's theorem yields 



p(5,m|2000) 



p(lep2| 1996, g, m)p{g, to|1996) 
p(lep2|1996) ' 



(A12) 



AlO and 



Here m|1996) is taken from Eq. 

p(lep2|1996) represents the density left in going from 
Fig. [iVc) to Fig. [TJd). The latter shows a plot of 
p{g, r7i|2000), showing that the region with m < 100 GeV 
has been zeroed. 

Letting m|2006) denote our understanding in 2006, 
comprising our knowledge in 2000 plus the additional 
knowledge (tev2a = tev(l fb~^)) of no signal beyond the 
Standard Model having been observed in Tevatron Run 
Ila, the posterior distribution ^(g, m|2006) obtained is 
shown in Fig. flte). The fraction of the density left in 
going from Fig^d) to Fig.jlje) is p(tev2a|2000) « 90%. 

Letting p{g, rn\2010) denote our understanding in 2010, 
comprising our knowledge in 2006 and assuming (tev2b = 
tev(10 fb^^)) that no processes beyond the Standard 
Model are observed in Tevatron Run lib, the posterior 
distribution p(g,m|2010) obtained is shown in Fig. IlTf). 
The fraction of the density left in going from Fig. Wfe) to 
Fig.[l|f) is p(tev2b|2006) « 88%. 

From the calculations above, in 2000 the expectation 
for finding new physics in Tevatron Run Ila was equal to 



(1 -p(tev2a|2000)) « 10%. 



(A13) 



In 2006, the expectation for finding new physics in Teva- 
tron Run lib is 



(1 -p(tev2b|2006)) « 12%. 



(A14) 



In 2000, the expectation for finding new physics in Teva- 
tron Run II is equal to 



(1 



(1 -p(tev2a,tev2b|2000)) 
]9(tev2a|2000) p(tev2b|2006)) 



20%. (A15) 



Carrying the argument forward, the expectation for find- 
ing new physics at the LHC is determined to be roughly 
90%. 

To recap, the physics content of the argument provided 
above is straightforward. Starting with a prior represent- 
ing the field's expectation in 1990 that the first sign of 
physics beyond the Standard Model will appear at or 
slightly above the electroweak scale, this prior is evolved 
through the experimental knowledge gained from Teva- 
tron Run I and LEP 2 to update our expectation to the 
year 2000. This expectation is then evolved through the 
experimental reality of Tevatron Run Ila and a hypo- 
thetical null result in Tevatron Run lib to find that the 
fraction of our expectation from 2000 that has been elim- 
inated by 2010 is - 20%. 

Of the roughly 2 x 10* kinematic distributions that can 
be considered in the Tevatron, it is difficult to convinc- 
ingly argue that any one of them is more likely to display 



the first sign of new physics than any other. Unless there 
are many signs of new physics in the Tevatron data, it 
therefore follows that any targeted search — including 
the search for single top production, Bg —^ fj,'^ ^~ , and 
the measurement of Bs mixing — has a discovery expec- 
tation of roughly 20%/2 x lO'^ = 10"^ 



2. fi2e 

The argument for the /i2e experiment is largely based 
on the observation that new electroweak scale physics 
with non-negligible lepton flavor violation can give rise 
to a transition fx e in the electric field of an atomic 
nucleus sufficiently large to be observable at fi2e. Of the 
new electroweak scale physics that may give rise to such 
a signal, electroweak scale supersymmetry is so far the 
most thoroughly studied possibility. 

A poll taken during a series of seminars over the past 
six years including over six hundred professional physi- 
cists [Hj suggests the field's expectation that weak scale 
supersymmetry is realized in nature is roughly 35%. 
An independent poll conducted online shows similar re- 
sults 30]. Interestingly, it is generally not the case that 
one third of the field is confident low energy supersym- 
metry is present, and that two thirds are confident low 
energy supersymmetry is not; informal discussions sug- 
gest most individual physicists would place the odds that 
the first sign of new physics will arise from low energy su- 
persymmetry at 2:1 against. 

Assuming low energy supersymmetry is realized, either 
there is significant mixing in the slepton sector (as in the 
case of neutrinos in the Standard Model) , or for some rea- 
son this mixing is suppressed (as in the case of quarks in 
the Standard Model). There being no particularly com- 
pelling reason to prefer either of these two scenarios over 
the other, an expectation of 50% can be assigned to there 
being significant mixing in the slepton sector, asssuming 
low energy supersymmetry is realized in nature. 

Assuming ^2e operates as designed, the expectation 
that a /i — *■ e signal identifying physics beyond the Stan- 
dard Model will be seen is thus roughly 35% x 50% — 
18%. 

Reasonable variations in this argument, including ex- 
panding the consideration of new physics scenarios that 
may lead to a /i — > e signal beyond electroweak scale 
supersymmetry, can lead to a final number differing by 
roughly a factor of two either way. The important point 
is that the discovery expectation cannot reasonably be 
argued to be significantly less than ~ 10% or greater 
than - 90% 05] . 

For a proposed experiment whose outcome is binary, a 
prior expectation of roughly 50% corresponds to a scien- 
tific merit of 0.30, and the experiment provides a full bit 
of information. A proposed experiment with binary out- 
come with a prior expectation to see a signal of 10% (or 
90%) has a scientific merit of 0.14. The scientific merit 
of the proposed fi2e experiment is significantly less than 
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the LHC experiments in part due to the relative inability 
of the ij2e experiment to further explore the underlying 
physics responsible for producing the signal. 

3. Historical discoveries 

The surprises corresponding to the discoveries of the c, 
T, 6, W , Z , and t particles have been estimated by com- 



munication with individuals active (both intimately and 
as observers) during the time of the discoveries. Written 
documents that have been helpful in this regard include 
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for its intuitive interpretation. 
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X — {xi, . . . , Xi, . . . , 2;ioo}, each of which has equal ex- 
pectation, so that p{xi) = 0.01 for all i £ {1, . . . , 100}. 
Obtaining the result X77 corresponds to a surprisal of 
surprisal(a::77) = — logjQ 0.01 — 2. Consider a third ex- 
periment corresponding to a collaboration of the first and 
second experiments with one thousand possible outcomes 

X = {(xi,xi), . . . , (a;i, a;ioo), (a;2,a;i), . . ■ , (sio, a;ioo)}. 
Obtaining the joint result (2:7, 2:77) corresponds to a sur- 
prisal of surprisal(x7, 2:77) — — log j^q 0.001 = 3 = 1-1-2, 
as desired. 

[33] The generalization to a continuous set of possibilities is 
straightforward. 

[34] Standard Bayesian notation is used. The joint probability 
p{xi,yj) is a number between zero and unity quantifying 
the degree of belief that the outcome of the experiment 
will be Xi and that the true hypothesis is j/j . The condi- 
tional probability p{yj\xi) is a number between zero and 
unity quantifying the degree of belief that the true hy- 
pothesis is yj , given that the outcome of the experiment 
is Xi. 

[35] Terms in the sum over j for which p(yj\xi) = vanish. 

[36] Since high energy physicists may find the usual no- 
tations for the Kullback-Leibler divergence, such as 
DKL{p{Y\xi)\\p{Y)), to be somewhat cryptic, we have 
opted to spell out "information gain" in equations, in 
the interest of clarity. 

[37] Proof: 

X]iP(^0 information gain (xi) = 
EiP(a^O Ej f(yi|a;i)evidence(yj|xi) = 

E.p(^0 E,p(y.l^O iogio1gr = 

E„,p(x.,y,)log,o'^ = 

T,i,jPixi,yj) [logioP(%|a;i) - log^oPiVj)] = 
-Ei,iP(a;i,J/j)logioP(yj)+EijP(2;»,yi) login p(yj>0 = 
-j:,P{y,)^og,,piy,) - H{Y\X) = 
H{Y) - H{Y\X) = AH. 
[38] This experiment is proposed to measure (or bound) the 



transition fi —> ein the electric field of an atomic nucleus. 
The iJ,2e co llaboration URL is currently maintained at 
http://mu2e.fnal.gov 

The style of scientific notation here and below (writing 
2e-01 rather than 2 x 10~^) is used to emphasize the 
exponent. 

To clarify notation: j/i — "exists" refers to the state 
of knowledge in which the Higgs boson is known to 
exist; y2 = "lexists" refers to the state of knowledge 
in which the Higgs boson is known not to exist; and 
Y = "exists?" refers to the two possible states of knowl- 
edge yi and j/2- Similarly, xi — "observed" refers to 
the experimental outcome in which the Higgs boson is 
observed; X2 = "lobserved" refers to the experimental 
outcome in which the Higgs boson is not observed; and 
X — "observed?" refers to the two possible experimental 
outcomes Xi and X2- 

To again clarify notation: yi = "ckm" refers to the state 
of knowledge in which the CKM picture is sufficient; 
2/2 ~ "!ckm" refers to the state of knowledge in which 
the CKM picture is insufficient; and Y = "ckm?" refers 
to the two possible states of knowledge j/i and y2- Simi- 
larly, xi — "observed" refers to the experimental outcome 
in which mixing is observed, and Arris is found to be 
< 30 ps~^; X2 = "lobserved" refers to the experimental 
outcome in which Bs mixing is not observed, and Arris 
is found to be > 30 ps~^; and X — "observed?" refers to 
the two possible experimental outcomes xi and X2- 
Such an estimation of incremental cost is inherently prob- 
lematic due to ambiguities in accounting. Incremental 
cost for analyses within "existing" experiments is taken 
to be given by the total number of person-years required 



for the analysis, at an estimated cost of lO'^ k$ per person- 
year. Incremental costs for analyses representing the pri- 
mary justification for an experiment's construction are 
taken to be the total cost of the experimental apparatus. 
[43] Although nothing beyond the Standard Model was dis- 
covered at Tevatron Run I or LEP 2, it should be noted 
that the number of published distributions comparing 
data to Standard Model prediction in both of these col- 
lider runs represents only roughly 1% of the distributions 
that might plausibly be expected to contain the first hint 
of new physics. For the purpose of this discussion (but 
contrary to fact), we assume the remaining distributions 
were all carefully considered and no anomalous effect was 
observed. 

[44] The seminar audience is asked whether the first sign of 
new physics will come from heavy gauge bosons, techni- 
color, extra spatial dimensions, leptoquarks, supersym- 
metry, a fourth generation of fermions, compositeness, 
or something else. The options are provided on a trans- 
parency, and audience members are requested to raise 
their hand as the above possibilities are read in sequence. 
The fraction of audience members who raise their hand 
for something, corresponding to a response rate for this 
poll, is roughly 50%. 

[45] The author's initial assessment of the probability of a 
signal being observed at fi2e was ~ 10"^ for reasons that 
did not withstand interrogation. This is an example of a 
case in which an individual's assessment of the scientific 
merit of an experiment has changed qualitatively through 
forced exposition of quantitative argument. 



